Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-offs by Selective Execution

We introduce Dynamic Deep Neural Networks (DNN), a new type of feed-forward deep neural network that allows selective execution. Given an input, only a subset of DNN neurons are executed, and the particular subset is determined by the DNN itself. By pruning unnecessary computation depending on input, DNNs provide a way to improve computational efficiency. To achieve dynamic selective execution, a DNN augments a feed-forward deep neural network (directed acyclic graph of differentiable modules) with controller modules. Each controller module is a sub-network whose output is a decision that controls whether other modules can execute. A DNN is trained end to end. Both regular and controller modules in a DNN are learnable and are jointly trained to optimize both accuracy and efficiency. Such training is achieved by integrating backpropagation with reinforcement learning. With extensive experiments of various DNN architectures on image classification tasks, we demonstrate that DNNs are general and flexible, and can effectively optimize accuracyefficiency trade-offs.